An interaction of objects with one another in an environment given specified starting conditions, can generally be referred to as a scenario. Various parameters of a system, such as robustness, can be categorized based on outcomes (i.e., the end result) of the scenarios given input from the system. Often, physical testing or validation of a particular scenario is neither viable nor possible. For sufficiently simple scenarios, it can be possible to obtain analytical solutions for testing or validation purposes, incorporating physical approximations where necessary. However, where situations become too complex, analytical solutions may not be viable. In such situations, testing and validation is done through the use of simulations.
As one example, simulations have been used to test operating software for autonomous vehicles. Simulation is required because it is impossible to safely test the software among traffic and pedestrians. Additionally, it is virtually impossible to repeat such tests using exactly the same conditions. For example, it would be nearly impossible to precisely recreate a situation in which an autonomous vehicle avoids a drunk driver on a highway over and over due to both safety concerns and repeatability. Moreover, physical testing also presents the limitation of the practicable number of scenarios which can be tested (e.g., it may not be feasible to test some result millions of times). These simulations, therefore, attempt to recreate the world, and obstacles within it, so as to accurately reproduce scenarios which can be encountered by an autonomous vehicle.